In the 1800s, the Irish potato famine droves millions of immigrants to America, where they settled in the worst part of New York City, the Five Points. Known as a center of vice and debauchery throughout the nineteenth century, Five Points was the neighborhood named for the points created by the intersection of Park, Worth, and Baxter streets. With no money and few prospects, many Irishmen turned to a life of crime. The first gangs had colorful nicknames like the "The Forty Thieves," "The Dead Rabbit Gang," and the "Plug Uglies." Corrupt politicians quickly learned that they could control blocks of voters—and elections—by buying gang support. Tammany Hall built its strength on the muscle of the Irish gangs.

Irish gangs fought the anti-immigrant hoods "The Bowery Boys" for control of the streets, resulting in several deadly riots, the most notable being the "Dead Rabbit Riot" of 1857, a street battle that left over a hundred dead. Five years later, the gangs of the Five Points became involved in the Draft Riots of 1863. The violence stemmed from protests that slum dwellers were being drafted in disproportionate numbers to fight for the Union. New York City burned for five days. Over a hundred buildings were destroyed and many more looted. Casualties were estimated at 2,000 dead, more than the Battle of Bull Run.

By the start of the twentieth century, Jewish organized crime took over the Five Points. The Jewish gangs hired themselves out as enforcers and bullied both labor unions and management during the labor unrest of the 1900s.

The new influx of Italian families in the early 1900s bred what became the most dominant group in American crime. Paul Kelly, a.k.a. Paolo Antonio Vacarelli, formed a mixed ethnic gang called "The Five Pointers." From this gang, and its farm team, "The Five Points Juniors," came some of the most infamous names in American crime—Al Capone, Lucky Luciano, Meyer Lansky, and Bugsy Siegel.